Image Features Based on Two-dimensional FFT for Gesture Analysis and Recognition

This paper describes features and the feature extraction processing which were applied for recognising gestures by artificial neural networks. The features were applied for two cases: time series of luminance rate images for hand gestures and time series of pure grey-scale images of the facial mouth region. A focus will be on the presentation and discussion of the application of 2dimensional Fourier transformed images both for luminance rate feature maps of hand gestures and for greyscale images of the facial mouth region. Appearancebased features in this context are understood as features based on whole images, which perform well for highly articulated objects. The described approach was used based on our assumption that highly articulated objects are of great interest for musical applications. I. SPATIAL APPEARANCE BASED FEATURES A. Feature Maps of Luminance Rate (Optical Flow Grade Zero) The visual energy of two consecutive video frames may be used as a feature map for the recognition of a visual time series such as hand gestures. This approach is understood as luminance rate. Different denotations are used for the luminance rate feature, such as optical flow of grade zero [23], difference image or visual energy. This has been used in several approaches both in scientifictechnical and musical/artistic environments [31], [32]. Advantages of this approach are: • Masking the (stationary) background • Robustness against variations of lighting such as: o Intensity o Contrast o Light Temperature • There are some suggestion that the approach is close to biological mechanisms (i.e. the high attention to motion in the visual cortex) • Fast to compute The drawbacks are: • Motion in the background scene detracts from the observed object • Related motions detract from the target (e.g. a motion of the hand is often combined with a motion of the whole arm) • Luminance rate is zero for still objects • There is a lower significance for slow moving objects Fig. 1: Tracked left hand with attention rectangle (image size 640x240) Fig. 2: Luminance rate feature for attention rectangle of Fig. 1 (frame size 160x80) Fig. 3: Luminance rate feature for attention rectangle of Fig. 1 (down-sample dimensions 32x32) A modified Cam-Shift algorithm may be used to focus the attention on the relevant part of the video stream i.e. the hand as shown in Fig. 1. For the context of this work this region of attention is named the Attention Rectangle (AR) or Region of Interest (ROI) as used in OpenCV [23]. The luminance rate image may be computed from the difference of two consecutive grey scale images of the AR according to Eq. 1. LumaRate(xAR , yAR ,t) = Luma(xAR , yAR ,t) Luma(xAR , yAR ,t 1) Eq. 1: Computation of the Luminance Rate, xAR,yAR: spatial coordinates of the Attention-Rectangle B. Feature Maps of pure Grey Scale Images (Mouth Gestures) Images of mouth gestures were used to investigate the recognition of static poses by recognition algorithms such as artificial neural networks. This was motivated by the prospect that, in addition to using energy as an intuitive control for musical parameters, position and force were both associable with stable gesture states. Furthermore gestures of the facial mouth region differ in the form of Proceedings SMC'07, 4th Sound and Music Computing Conference, 11-13 July 2007, Lefkada, Greece